Geodesic Learning Algorithms Over Flag Manifolds

Recently manifold structures have attracted attentions in two folds in the machine learning literature. One is in the manifold learning problem, that is learning the intrinsic manifold structure in high dimensional datasets. Another is in the information geometric approach to learning – exploiting the geometry of the parameter space of learning machines such as neural networks for improving conventional learning algorithms [1]. In this presentation we discuss the use of manifolds for the latter aim, particularly, we investigate the flag manifold arising from one layer neural networks for solving subspace ICA problems. When the parameter space of neural networks forms a manifold, the learning equation is defined over that manifold and should be integrated taking the manifold structure into account. The Riemannian learning method [3], by introducing a Riemannian metric into the manifold, yields a discretization scheme for solving differential equations on the manifold: integration of the differential equation is performed along piecewise geodesics which approximate the original learning trajectory. This Riemannian method was successfully applied to the orthogonal group, i.e. the Lie group of orthogonal matrices, for the non-negative ICA problem [5]. In the context of optimization this Riemannian approach was also considered over the Stiefel and the Grassmann manifolds in [2]. The Grassmann manifold just considers the set of subspaces of a fixed dimension, while the flag manifold consists of the set of the direct sum of subspaces and includes the Grassmannian manifold as a special case. Therefore the flag manifold can deal with several subspaces simultaneously and is suitable for tackling subspace ICA problems. We extend the formulas obtained in [2] to the flag manifold. The effectiveness of the Riemannian learning method over the standard Euclidean algorithms is illustrated in independent subspace analysis (ISA) and complex ICA experiments. Also, the issue of local minima in ISA is discussed. We combine the Riemannian learning method over the flag manifold and an MCMC method to overcome the problem of local minima of the ISA cost function.